Evaluating Large Language Models in Extracting Cognitive Exam Dates and Scores.

Importance: Large language models (LLMs) are crucial for medical tasks. Ensuring their reliability is vital to avoid false results. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Objective: Evaluate ChatGPT and LlaMA-2 performance in extracting MMSE and CDR scores, including their associated dates. Methods: Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss' Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. Results: For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT's errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. Conclusions: In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

Thrombogenicity of Hyaluronic Acid Fillers: A Quantitative Thrombodynamics Study.

PURPOSE: At present, there is a paucity of data regarding the thrombogenicity of hyaluronic acid fillers (HAFs). This article quantitatively analyses the thrombogenicity of 2 commonly used HAFs: Restylane Lyft and Juvéderm Ultra. METHODS: Thrombogenicity was assessed using the Thrombodynamics Analyzer System and plasma obtained from healthy controls. Following the addition of HAFs or control, spontaneous clot formation time, initial rate of clot growth, average rate of clot growth over 30 minutes, and clot size at 30 minutes was measured for each sample. The median of differences between each group were analyzed. RESULTS: Nine individuals with a mean (SD) age of 37 (17) years, participated in the study. Initial rate of clot growth was significantly lower in plasma mixed with Juvéderm compared to control (p = 0.008) or Restylane (p = 0.038). The average rate of clot growth more than 30 minutes was significantly lower in both HAF groups (Restylane vs. control p = 0.038; Juvéderm vs. control p = 0.008), there was no significant difference between HAF groups (p = 0.635). Final clot size was significantly smaller with Juvéderm (p = 0.038 vs. control and p = 0.013 vs. Restylane). Spontaneous clot formation time did not significantly change with the addition of either HAF. CONCLUSIONS: Juvéderm significantly reduces the initial rate of clot growth, the average rate of clot growth more than 30 minutes, and clot size, whereas the addition of Restylane decreases the average rate of clot growth without affecting overall clot size in healthy individuals.

Reversible Electrochemical Sensor for Detection of High-Charge Density Polyanion Contaminants in Heparin.

We present a simple, rapid, and inexpensive electrochemical sensor based on a reversible pulsed chronopotentiometric polyanion-selective membrane electrode for the detection and quantification of oversulfated chondroitin sulfate (OSCS) and other high charge-density polyanions that could potentially be used to adulterate heparin. The membrane is free of ion exchanger and is formulated with plasticized poly(vinyl chloride) (PVC) and an inert lipophilic salt, tridodecylmethylammonium-dinonylnaphthaline sulfonate (TDMA-DNNS). The neutral salt is used to reduce membrane resistance and to ensure reversibility of the sensor. More importantly, TDMA(+) is used as the recognition element for the polyanions. Here an anodic galvanostatic current pulse is applied across the membrane to cause the extraction of the polyanions from the sample into the membrane and potential is measured at the sample-membrane interface. The measured electromotive force (emf) is proportional to the concentration and the charge density of the polyanions. High charge-density polyanion contaminants and impurities in heparin can be detected using this method since the overall equilibrium potential response of polyions increases with increasing charge density of the polyions. Here, first the potential response of pure heparin is measured at a saturation concentration, the concentration beyond which further addition of heparin does not produce a change in potential response. Then the potential response of heparin tainted with different quantities of the high charge-density contaminant is measured at a fixed total polyion concentration (heparin concentration + contaminant concentration). The latter gives a greater negative potential response due to the presence of the high charge-density contaminant. The increase in the negative potential response can be used for detection and quantification of high charge-density contaminants in heparin. We demonstrate here that 0.3% (w/w) OSCS as well as 0.1% (w/w) dextran sulfate can be detected in heparin at 20-mg/mL total polyion concentration. It has also been shown that 1% (w/w) of dextran sulfate can readily be detected in heparin at only 2-mg/mL total polyanion concentration with a linear response (R(2) = 0.994).